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Abstract 

We introduce a novel method for using reflectance to 
identify materials. Reflectance offers a unique signature of 
the material but is challenging to measure and use for rec¬ 
ognizing materials due to its high-dimensionality. In this 
work, one-shot reflectance is captured using a unique op¬ 
tical camera measuring reflectance disks where the pixel 
coordinates correspond to surface viewing angles. The re¬ 
flectance has class-specific stucture and angular gradients 
computed in this reflectance space reveal the material class. 
These reflectance disks encode discriminative information 
for efficient and accurate material recognition. We intro¬ 
duce a framework called reflectance hashing that models 
the reflectance disks with dictionary learning and binary 
hashing. We demonstrate the effectiveness of reflectance 
hashing for material recognition with a number of real- 
world materials. 


1. Introduction 

Color and geometry are not a full measure of the richness 
of visual appearance. Material composition of a physical 
surface point determines the characteristics of light interac¬ 
tion and the reflection to an observer. In the everyday real 
world there are a vast number of materials that are useful 
to discern including concrete, metal, plastic, velvet, satin, 
asphalt, carpet, tile, skin, hair, wood, and marble. A com¬ 
putational method for identifying these materials has im¬ 
portant implications in developing new algorithms and new 
technologies for a broad set of application domains. For ex¬ 
ample, a mobile robot or autonomous automobile can use 
material recognition to determine whether the terrain is as¬ 
phalt, grass, gravel, ice or snow in order to optimize me¬ 
chanical control. An indoor mobile robot can distinguish 
among wood, tile, or carpet for cleaning tasks. The material 
composition of objects can be tagged for an e-commerce 
inventory or for characterizing multi-composite 3D printed 
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Figure 1: Reflectance disks provide a quick snapshot of the 
intrinsic reflectance of a surface point. Gradients of the re¬ 
flectance space are captured with textons and provide a sig¬ 
nature for material recognition. 

objects. The potential applications are limitless in areas 
such as robotics, digital architecture, human-computer in¬ 
teraction, intelligent vehicles, and advanced manufacturing. 
Furthermore, just as computer vision algorithms now use 
depth sensors directly from RGB-D cameras, material sen¬ 
sors can have foundational importance in nearly all vision 
algorithms including segmentation, feature matching, scene 
recognition, image-based rendering, context-based search, 
object recognition, and motion estimation. Our approach 
uses reflectance for material recognition because of the ad¬ 
vantages of having an intrinsic optical signature for the sur¬ 
face. However, we bypass the use of a gonioreflectometer 
by using a novel one-shot reflectance camera based on a 
parabolic mirror design. The output of this camera is a re¬ 
flectance disk, a dense sampling of the surface reflectance of 
the material projected into a single image as shown for two 
example surfaces in Figure 1 . Each reflectance disk mea- 
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Figure 2: Reflectance disks provide a snapshot of the re¬ 
flectance function for spot samples on the surface. For this 
example of a peacock feather, iridescence causes a large 
variation of intensity with viewing direction as revealed in 
the three reflectance disks. 


sures surface properties for a single point and can capture 
complex appearance such as the iridescence of a peacock 
feather as illustrated in Figure 2. We then use this conve¬ 
nient reflectance measurement as the discriminative charac¬ 
teristic for recognizing the material class. We address the 
issue of high dimensionality using a novel application of 
binary hash codes to encode reflectance information in an 
efficient yet discriminitve representation. The key idea is 
to obtain sufficient sampling of the reflectance with enough 
discriminative power and reduce its representation size so 
that it can be effectively used for probing the material. 

We present a database of reflectance images comprised 
of twenty different diverse material classes including wood, 
velvet, ceramic and automotive paint with 10 spot measure¬ 
ments per surface and with three different surface instances 
per class. Measurements include three on-axis illumination 
angles (—10°, 0°, 10°) and ten random spot measurements 
over the surface. Each spot measurement is a reflectance 
disk composed of a dense sampling of viewing angles total¬ 
ing thousands of reflectance angles per disk. The database 
of 3600 images or reflectance disks is made publicly avail¬ 
able. For recognition, we combine binary hash coding 
and texton boosting for a new framework called reflectance 
hashing for efficient and accurate recognition of materials. 
We compare reflectance hashing with texton boosting for 
the task of recognizing materials from reflectance disks. 

2. Related work 

Prior methods for material recognition use two dis¬ 
tinct approaches. One approach assesses material iden¬ 
tity using reflectance as an intrinsic property of the surface 


Figure 3: Schematic of the mirror-based camera. Re¬ 
flectance disks are obtained by viewing a single point un¬ 
der multiple viewing directions using a concave parabolic 
mirror viewed by a telecentric lens. 

[26, 5, 19, 17, 13]. Another main approach identifles ma¬ 
terial labels using the appearance of the surface within the 
real world scene [34, 23, 21]. Using reflectance instead of 
scene appearance has the advantage that reflectance is a di¬ 
rect measurement of the material characteristics, instead of 
its phenomenological appearance [ 1 ]. Reflectance is mostly 
unique to the material, whereas the appearance is the con¬ 
voluted end result of the interaction of all the intrinsic and 
extrinsic factors and thus more difficult to decipher. 

A challenge in using reflectance for material recogni¬ 
tion is that measurements are typically difficult and cumber¬ 
some. For example, most methods require knowledge of the 
scene like geometry [26, 27] or illumination [31, 30]. Other 
methods require lab-based measurements of the BRDF 
(bidirectional reflectance distribution function) or BRDF 
slices such as light domes for illumination patterns [22]. 
Acquiring full BRDF requires gonioreflectometers that are 
comprised of multiple cameras and light sources covering 
the hemisphere of possible directions using geodesic domes 
or robotics as in [9, 43, 20, 8, 28, 29, 42]. Surface appear¬ 
ance in terms of relief texture is captured by the GelSight 
camera [15] using three-color photometric stereo by im¬ 
printing the surface geometry on an elastomer surface with 
Lambertian skin. With this device very flne geometric tex¬ 
ture can be recovered but surface appearance is lost. An ad¬ 
ditional challenge of using reflectance for measurements is 
the high dimensionality, reflectance as a function of illumi¬ 
nation and viewing direction especially if densely sampled 
can lead to thousands or more samples per surface point. 

Specialized cameras have been developed in prior re¬ 
search to obtain reflectance measurements of surfaces ef- 
flciently. The mirror-based camera illustrated in Figure 7 
is one such a device [8, 6]. This camera views an image 
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Figure 4: Gradients on the Reflectance Disk Extracted with Textons: Images are Altered by a Alter bank comprised of 
Gaussian, Laplacian and oriented gradient Alters. The 24 x 1 responses vectors are clustered to form visual words or textons. 
A random subsampling of rectangular subregions of the reflectance space deflnes regions of interest r. The feature of interest 
is 5'(r, t), the count of pixels identifled as texton t in region r. 


of a single surface point by using a concave parabolic mir¬ 
ror with the focus coincident with the target point. In this 
manner, the camera records a multiview image of a surface 
point where each pixel records a different angle. Multi¬ 
ple views from mirror-based optics is also achieved with 
a kaleidoscope-like conflguration [14]. However, the view¬ 
ing angles captured in this device are discrete and sparse. 
Because our approach relies uses gradients in angular space 
a dense sampling of reflectance is needed and we use the 
parabolic mirror-based camera as a reflectance sensor. 

In prior work, recognition of standard scene images is 
typically accomplished using image features that capture 
the spatial variation of image intensity. For example, im¬ 
age Altering with a set of Alter banks followed by grouping 
the outputs into characteristic textons has been used for im¬ 
age texture [19, 4, 40], object recognition [35, 36] and im¬ 
age segmentation [18]. Similarly, we encode the discrim¬ 
inative optical characteristics of materials captured in the 
reflectance disks with a texton-based representation. Tex¬ 
ton methods for the general problem of scene and image 
recognition have been improved signiflcantly by incorpo¬ 
rating boosting as in Texton-boost [36] wd Joint-boost [38]. 
We incorporate the utility of boosting to identify a discrim- 
initive and compact representation. 

The goal of efficient recognition of images for large scale 
search has led to numerous methods for binary hash codes 
[16, 2, 41, 24, 25]. A common approach in recent work 
[44, 37, 16, 33] is learning binary codes where the Ham¬ 


ming distance approximates the kernel distance between ap¬ 
pearance descriptors. This method is used when the ker¬ 
nel distance is known to give good performance but is ex¬ 
pensive to compute and store. Inspired by the work in 
image search, we develop reflectance hashing with binary 
codes that allow matching of measured reflectance disks 
with those in the labeled training set. The binary codes are 
learned using the match metric obtained from the texton- 
based descriptor we present in Section 3. We use a suite of 
state-of-the-art hashing techniques applied to our measured 
reflectance disk dataset to evaluate recognition performance 
in terms of speed and recognition accuracy. 

3. Methods 

3.1. Reflectance Disk Measurements 

We use the texture camera approach introduced in [7, 6] 
for measuring reflectance of surface points. The measure¬ 
ment camera mainly consists of a parabolic mirror, a CCD 
camera, a movable aperture and a beam splitter. The imag¬ 
ing components and their arrangement are illustrated in Fig¬ 
ure 3. A parabolic mirror section is flxed so that its focus is 
at the surface point to be measured. The illumination source 
is a collimated beam of light parallel to the global plane of 
the surface and passing through a movable aperture. The 
angle of the incident ray reaching the surface is determined 
by the intersection point with the mirror. Therefore, the il¬ 
lumination direction is determined by the aperture position. 


























Figure 5: Gradients on the reflectance disk are computed us¬ 
ing image Altering. However, the angular resolution varies 
over the surface as shown above. The angle at ab is differ¬ 
ent from cd. Therefore differences (or Altering) computed 
with a uniform spacing on the reflectance disk have a non- 
uniform mapping to angles. 


Light reflected from the surface point is reflected by the mir¬ 
ror to a set of parallel rays directed through a beam splitter 
to the camera. Each pixel of this camera image corresponds 
to a viewing direction of the surface point. Therefore, the 
recorded image is an observation of a single point on the 
surface but from a dense sampling of viewing directions. 
Multiple illumination directions can be measured by pla¬ 
nar motions of the illumination aperture. A key advantage 
of this approach is the potential for a handheld reflectance 
sensor where the mirror and light source are attached to a 
handheld camera. 

We refer to the camera’s output images as reflectance 
disks since they depict circular regions from on-axis projec¬ 
tions of a paraboloid. Examples of these reflectance disks 
are shown in Figure 1 for two surfaces. These examples 
show the variation of the surface reflectance with viewing 
angle. Reflectance disks have an intensity variation that 
corresponds to angular change with the viewing direction 
instead of spatial change on the surface. However, the re¬ 
flectance disks can be interpreted as images and Altered to 
And characteristic gradients. This approach of using angu¬ 
lar gradients in reflectance space is a novel contribution of 
our work. Our goal is to represent the measurements in an 
efficient and meaningful way that supports material recog¬ 
nition. 

3.2. Textons on the Reflectance Disk 

We And characteristic patterns of intensity in a re¬ 
flectance disk by computing spatial change in the re¬ 


flectance disk via image Altering. Since spatial coordinates 
of the reflectance disk map to angular coordinates of re¬ 
flectance, Altering is a convenient way to compute angular 
gradients of the reflectance. A common method of quanti¬ 
fying intensity patterns is using textons. Textons are com¬ 
puted by first Altering the image with a set of gradient Alters 
at different orientations and scales. We employ the same Al¬ 
ter banks as in [19], comprised of Gaussian low-pass Alters 
at four scales, Laplacian Alters at four scales and eight gra¬ 
dient Alters at different orientations and scales for a total 
of 24 Alters. The resulting 24 x 1 responses at each pixel 
are clustered using iT-means clustering and a dictionary of 
visual words called textons is created. The underlying as¬ 
sumption is that the local intensity variation can be captured 
with the finite set of characteristic Alter responses that are 
the centers of each of the K clusters. 

We use textons as a reflectance feature to provide a 
dense, per-pixel description of reflectance variation. Meth¬ 
ods to detect key-points are useful for scene images, but 
reflectance disks do not typically have specific key-points 
of interest. 

For texton computation, we use gradient Alters that ap¬ 
proximate the derivative operator with a discrete spatial dif¬ 
ference Alter. This discrete approximation of the gradient 
computes change over a non-infinitesimal distance, typi¬ 
cally the difference of neighboring pixels. The distance 
between these neighboring pixels is constant over the re¬ 
flectance disk, i.e. the same gradient Alter is applied over the 
entire disk image. However, because of the mapping from 
spatial coordinates to angular coordinates by the parabolic 
mirror, the angular distance is not constant. This situation 
is illustrated in Figure 5 where the cone angle at the surface 
is shown for two different locations on the parabolic mir¬ 
ror. A constant distance d on the projected disk leads to two 
different cone angles. Consequently, the resolution for the 
angular gradient varies as a function of disk coordinates. 

The equation of the parabolic mirror surface is given by. 


y + F = 


4F ’ 


( 1 ) 


where the ^-axis is the optical axis of the mirror aligned 
with the camera optical axis, the x — z plane is aligned with 
the disk-shaped projection of the mirror surface on camera’s 
image plane, and F is the mirror focal length (12.7 mm). 
Define the cone angle a as the set of angles subtended by arc 
ab or cd in Figure 5, where d is taken as 2 pixel units. From 
this equation, the variation of the cone angle a as a function 
of the vertical spatial coordinate 2 ; on the reflectance disk is 
derived in [6] as 
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Since the geometry of the mirror is known, the spatial Al¬ 
ters can be adaptive to ensure a uniform cone angle over the 











disk. However, the spatially invariant gradient filter pro¬ 
vides a discriminitive representation and uniformity of an¬ 
gular resolution is not required. 

Reflectance-Layout Filter Textons are a local measure 
of image structure, and texton histograms are used to rep¬ 
resent a region. The histogram representation has the ad¬ 
vantage of invariance to shifts of structures. Reflectance 
disks show the structure of the reflectance field, but of¬ 
ten the distribution within that structure is the characteris¬ 
tic property. Histogram representations have the drawback 
that crucial spatial information is lost. We follow the texton 
boost framework [35, 36, 38] to encode the characteristic 
spatial structure of the reflectance disk by defining a set of 
randomly shaped rectangular regions over the disk. For a 
given texton t and region r, S{r,t) is the count of pixels 
labelled with texton t that are located within region r as il¬ 
lustrated in Figure 4. We use this response as a key feature 
for describing the reflectance disks. This response is re¬ 
ferred to as the texture-layout filter response when applied 
to ordinary images [36]. For material recognition using re¬ 
flectance disks, we use the term reflectance-layout filter to 
indicate that the representation captures the angular layout 
of refiectance gradients. For computing refiectance-layout 
filter responses, the number of particular textons in a re¬ 
gion, the texton count is calculated using a soft-weighting 
for the 8 nearest cluster centers. Soft-weighted textons as in 
[39, 36, 18] allows each pixel to be associated with 8 near¬ 
est clusters instead of a single texton label. This method 
alleviates the problem of two similar responses assigned to 
different texton labels. 

Each refiectance layout filter characterizes a specific re¬ 
gion in refiectance space where the change of refiectance as 
a function of viewing angle is a descriptive feature for the 
material. There are many possible combinations of r, t as 
illustrated in Figure 4 and we want to choose the ones that 
are most discriminitave. The boosting approach in [35, 38] 
is used to identify the most discriminitive r, t combinations. 
After boosting, the typical numbers of refiectance layout fil¬ 
ters is approximately 700. 

3.3. Boosting 

The algorithm for Joint Boosting [38] provides feature 
selection where the feature is the reflectance-layout filter 
5'(r, t) specified by a particular (r, t) pair that indicates the 
number of times texton t occurs in region r. The approach 
builds a strong learner iteratively by adding a strongest 
weak learner which is the selected feature in each itera¬ 
tion, allowing the weak learner to help classify a subset of 
classes. At each step m, the method maximally reduces 
the weighted squared error by choosing a strongest weak 
learner updating strong learner as: 77(/^,c) := 

77(/i, c) + hm{h^ c), where li is the input image and c is 


the class. The weighted square error is given by 

C N 

Jwse = - hm{Ii,c))‘^ (3) 

c=l i=l 

where C is number of classes, and N is the number of train¬ 
ing samples, defines the weights for class 

c and sample i that emphasize the incorrectly classified sam¬ 
ples for that iteration, is the membership label (±1) for 
class c. The optimal solution that gives the selected feature 
at step m has the form 

ad{S{rfl) > 0) if c G 

otherwise 

with the parameters {a^b^k^^O) given by the minimization 
of Equation 3, (5(.) is indicator function (0 or 1), S{rfl) 
is the count or texton t in region r, is the class subset 
that share this feature. For the classes sharing the feature 
{ci G Tm), the weak learner gives h{Ii,c) G {a + 6,6} 
determined by the comparison of reflectance-layout filter 
response S{rfl) with the threshold 9. For the classes not 
sharing the feature, there is a constant k^ different for each 
class c that avoids the asymmetry caused by the number of 
positive and negative samples for each class. 

This method chooses a dominant feature at 

each iteration to minimize the error function. For each se¬ 
lected feature, finding the class subset T that maximally 
reduces the error on weighted training set is expensive. 
Instead of searching all possible 2^ — 1 combinations, a 
greedy search heuristic [38] is used to reduce the complex¬ 
ity to O(C^). We first find the best feature for each single 
class, and pick the class that has the best error reduction. 
Then we add the class that has best joint error reduction un¬ 
til we go through all the classes. Finally, we select the set 
that provides the largest overall error reduction. It is also 
time consuming to go through all the features at each itera¬ 
tion, so only a fraction r 1 reflectance-layout filters are 
examined, randomly chosen in each iteration. 

3.4. From Textons to Binary Material Codes 

Boosting allows the selection of features that 

specify r, t (region, texton) pairs of interest. The selected 
features can simply be viewed as the input feature vector for 
a basic nearest neighbor classification. Nearest neighbors in 
high dimensions is problematic, but the computational tools 
of binary hash codes enables efficient and accurate repre¬ 
sentation. We use the reflectance layout filters directly in a 
concatenated feature vector with 700 x c component where 
each component is a region-texton r, t pair that is known 
to be useful in material recognition. There is likely to be 
redundancy in the r, t combinations so the number of com¬ 
ponents in the feature vector may be reduced accordingly. 



Figure 7: Prototype of the camera for reflectance disk cap¬ 
ture. The components follow the arrangement in Figure 3. 


This high dimensional feature vector is used directly for fast 
computation by employing binary coding. This efficiency 
is highly desirable for the embedded hardware implementa¬ 
tion. 

We use a suite of state-of-the-art approaches for form¬ 
ing binary code words including circulant binary em¬ 
bedding (CBE) [45], bilinear embedding [10], iterative 
quantization (ITQ) [12], angular quantization-based binary 
codes(AQBC) [11], locality sensitive hash(LSH) [3], and 
locality-sensitive binary codes from shift-invariant kernels 
(SKLSH) [32]. 

4. Experiments 

Reflectance Disk Database To evaluate the perfor¬ 
mance of our refiectance hashing framework, a database of 
3600 refiectance images is collected consisting of the fol¬ 
lowing 20 surface classes: cardboard, CD, cloth, feather, 
textured rubber, glossy ceramic, glossy metal, leather, linen, 
mattel metal, automotive painting, non-ferrous metal, pa¬ 
per, plastic, rock, rubber, sponge, velvet, glossy paper and 
wood. The measurement camera shown in Figure 7 is 
arranged as in Figure 3 with a video camera, telecentric 
lens, light source, beam splitter and parabolic mirror. The 
database includes 3 instances per class, i.e. three different 
example surfaces per class, 10 spot samples per surface with 
3 illumination directions ( — 10°, 0,10°) where 0° is frontal 
illumination aligned with the surface normal. Addition¬ 
ally, two exposure settings are collected for each refiectance 
disk. Therefore, the dataset consists of 180 refiectance disk 
images per class, for a total dataset of 3600 refiectance disk 
images each of size 320 x 240. This refiectance dataset is 
made publicly available (link provided in final publication). 

Material Recognition We compare the performance of 
texton boosting with refiectance hashing for material classi¬ 



Figure 8: Recognition rate as a function of training set size 
for both boosting and refiectance hashing. Notice the per¬ 
formance of refiectance hashing is high even when the train¬ 
ing set number is low. 


fication. We compared the classification result with a train¬ 
ing set of N random selected images for both methods, and 
test the classification accuracy on a test set of 3600 — N ran¬ 
dom selected images. We vary N from 360 (10%) to 1500. 
For the large set of training images, both methods perform 
well; however, the recognition rate of joint boosting de¬ 
creases significantly, when decreasing the size of training 
set. Refiectance hashing gives a more stable recognition 
rate as a function of the training set size as shown in Figure 
8 . 

Figure 9 shows the confusion matrix from N = 360 
classification with joint boosting with 700 iterations in the 
training stage, and mean recognition rate of 84.38%. The 
parameter settings were = 512 textons and 200 sub win¬ 
dows or regions, with the random feature selection fraction 
r = 0.01. Figure 10 shows the confusion matrix obtained 
by our method of refiectance hashing, with the features se¬ 
lected in 700 iterations, and using iterative quantization [12] 
as the binary embedding method. The overall recognition 
rate is 92.3%, and several individual class recognition rates 
are significantly higher than the boosting method of Fig¬ 
ure 9. For both methods, 5-fold cross validation was per¬ 
formed. 

We also make a baseline comparison with texton his¬ 
togram classification where no boosting is done and no re¬ 
gion subwindows are used. The histogram is computed over 
the entire refiectance disk. The recognition rate with 1500 
training images is 79.53% with a max rate of 100% (plas¬ 
tic, glossy paper) and a minimum of 45.6% (sponge) The 
recognition rate with 360 training images is 41.94% with a 
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Figure 6: Visualizing the reflectance-layout Alters.The first row shows the reflectance disks for different materials. The second 
row shows the corresponding texton maps. The third row shows three of the dominant region-texton (r, t) combinations 
chosen for each class. 


max rate of 89.2% (cardboard) and a minimum of 18.3% 
(linen). 

From the empirical results, we see that reflectance hash¬ 
ing provides reliable recognition even for a small training 
set. This result has important implications for real time 
material recognition since the approach may support online 
training with compact training sets. 

Evaluation of Hash Codes We compare the perfor¬ 
mance of recent binary code methods for use in reflectance 
hashing. These codes are use to represent the response from 
the reflectance-layout Alters as described in Section 3.4. We 
test the following binary embedding methods: randomized 
(CBE-rand) and learned (CBE-opt) versions of circulant bi¬ 
nary embedding (CBE) [45], randomized (BITQ-rand) and 
learned (BITQ-opt) versions of bilinear embedding [10], it¬ 
erative quantization (ITQ) [12], angular quantization-based 
binary codes(AQBC) [11], locality-sensitive binary codes 
from shift-invariant kernels (SKLSH) [32], and a baseline 
method locality sensitive hash (LSH) [3]. We used the 
publicly available software implementation for these binary 
embedding methods [45]. Recognition is accomplished us¬ 
ing a nearest neighbor search of binary codes with a Ham¬ 
ming distance metric and ten nearest neighbors. From Fig¬ 
ure 11, we can see the binary embedding recognition rate 
reaches around 90%, when using 128 or 256 bits of bi¬ 
nary codes. The method of ITQ gives the best results for 
this material recognition task, and we use this binary code. 
The CBE-opt and BITQ-opt do not work well at low dimen¬ 
sional case, but the recognition rate increases quickly with 
the number of bits in the code. 


5. Summary and Conclusion 

We have introduced a novel framework for measurement 
and recognition of material classes. The approach encodes 
high dimensional reflectance with a compact binary code. 
We have compared several existing binary code methods 
to choose the most appropriate for this material recogni¬ 
tion aproach. The coding supports discrimination among 
the classes and can be realized in embedded vision imple¬ 
mentations. Our method of reflectance hashing is com¬ 
pared to two popular baseline texton-based methods, boost¬ 
ing and texton histograms for material recognition. The 
results show excellent performance even for small train¬ 
ing sets and provide a novel method based on reflectance 
for fast sensing and recognition of real-world materials. 
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Figure 9: Joint Boosting Classification Results. Confusion matrix with percentages row-normalized. Overall accuracy is 
84.38%. 
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Figure 10: Reflectance Hashing Classification Results. Confusion matrix with percentages row-normalized. Overall accuracy 
is 92.3%. 
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